The present application relates to an exposure control method and an exposure control apparatus to be used in an image pickup apparatus such as a video camera or a digital still camera including an XY address scanning type imaging element (imager, image sensor), which may typically be a CMOS (complementary metal oxide semiconductor) imaging element, under the lighting of a fluorescent lamp and also an image pickup apparatus.
When an image of a subject is picked up by means of a video camera in direct light from a fluorescent lamp energized by a commercial AC power supply, temporal fluctuations, or so-called flicker, occur in the lightness of the video signal output as a result of the image pickup operation due to the difference between the frequency of luminance change of the fluorescent lamp (change in the quantity of light), which corresponds to twice the frequency of the commercial AC power supply, and the vertical synchronizing frequency of the camera.
For example, when an image of a subject is picked up by a CCD camera of the NTSC system (with the vertical synchronizing frequency of 60 Hz) under the lighting of a non-inverter type fluorescent lamp in a geographical area where the frequency of the commercially supplied AC is 50 Hz, the exposure value of each pixel changes every field as shown in FIG. 1 of the accompanying drawings because of that a field period is 1/60 seconds while the period of luminance change of the fluorescent lamp is 1/100 seconds and hence the timing of exposure of each field is shifted relative to the luminance change of the fluorescent lamp.
Thus, referring to FIG. 1, when the exposure time is 1/60 seconds, periods a1, a2, a3 show different exposure values even if the exposure time is same. When the exposure time is shorter than 1/60 seconds (but not equal to 1/100 seconds for the reason as will be described hereinafter), periods b1, b2, b3 show different exposure values even if the exposure time is same.
The timing of exposure relative to the luminance change of the fluorescent lamp returns to the original one in every three fields and therefore the change of lightness due to the flicker is cyclic and repetitive with a period of three fields. In other words, the luminance ratio of each field (how flickering appears) changes with the exposure period but the period of flicker does not change.
However, the change of lightness in every three frames is repeated with progressive type cameras such as digital cameras when the vertical synchronizing frequency is 30 Hz.
Additionally, a number of different phosphors such as red, green and blue phosphors are normally used in a fluorescent lamp in order to emit white light. However, such different phosphors have respective afterglow characteristics and they emit attenuating light with their respective afterglow characteristics during the period from a stop of electric discharge to the next start of electric discharge that is found in a period of luminance change. Therefore, light that is white at the beginning attenuates, gradually changing the hue, in that period. Then, if the timing of exposure is shifted as described above, there arises not only a change of lightness but also a change of hue. Additionally, since fluorescent lamps have a specific spectral characteristic that a remarkable peak is found at a particular wavelength, the fluctuating components of a signal change from color to color.
Thus, a so-called color flicker phenomenon occurs due to such a hue change and the difference of the fluctuating components from color to color.
On the other hand, if the exposure time is defined to be integer times of the period of luminance change of the fluorescent lamp ( 1/100 seconds) as shown in the lowermost part of FIG. 1, no flicker occurs because a constant exposure value is realized regardless of the timing of exposure.
Actually, there has been proposed a system adapted to detect light being shed from a fluorescent lamp by means of a user's operation or a signal processing operation of the camera and, if light from a fluorescent lamp is detected, set the exposure time to integer times of 1/100 seconds. With such a system, it is possible to fully prevent flickering from taking place by means of a simple technique.
However, it is not possible to select a desired exposure time with such a system so that the degree of freedom of using an exposure adjusting unit of the camera is limited when trying to achieve an appropriate exposure.
Thus, there is a demand for techniques of reducing the flickering of light of a fluorescent lamp regardless of the selected shutter speed (exposure time).
When an image pickup apparatus is so designed that all the pixels are exposed to light at a same timing for an image as in the case of a CCD image pickup apparatus, such a technique can be realized relatively easily because the luminance change and the color change appear only among fields and not in each field.
Referring again to FIG. 1, the flicker changes cyclically and repetitively with a period of three fields if the exposure time is not integer times of 1/100 seconds so that it is possible to suppress the flicker to a level that practically does not give rise to any problem by predicting the current change in the luminance and the colors from the video signal of the field that precedes the current field by three fields and adjusting the gain of the video signal of each field according to the prediction in order to make the average value of the video signals of respective fields.
However, in the case of an XY address scanning type imaging element, which may typically be a CMOS imaging element, the timing of exposure of each pixel is sequentially shifted from that of the preceding pixel in the horizontal direction by a clock (pixel clock) period and hence the timings of exposure of all the pixels differ from each other. In other words, it is not possible to satisfactorily suppress the flicker by means of the above-described technique.
FIG. 2 schematically illustrates this problem. While the timings of exposure of all the pixels differ from each other in the horizontal direction as pointed out above, one horizontal period is sufficiently shorter than the period of luminance change of a fluorescent lamp. Therefore, it is assumed here that the timings of exposure of all the pixels on a horizontal line are same and FIG. 2 shows the timing of exposure of each of the lines arranged in the vertical direction. Such an assumption does not produce any problem in practical applications.
Referring to FIG. 2, in the case of an XY address scanning type imaging element such as a CMOS imaging element, the timing of exposure of each line differ from that of any other line (F1 in FIG. 2 indicates the difference) and hence the exposure value differ from line to line. Then, a luminance change and a color change due to the flicker take place not only among fields but also in each field so that the obtained image shows stripes (that run horizontally and repeat vertically).
FIG. 3 illustrates the flicker in an image when the subject is a uniform pattern. Since the cycle period (wavelength) of stripes is 1/100 seconds, a striped pattern of 1.666 cycle periods is produced in the image. If the number of lines read out per field is M, the number of lines of the striped pattern read out per cycle period is L=M* 60/100. Note that the asterisk (*) is used herein as symbol of multiplication.
As shown in FIG. 4, a striped pattern of five periods (for five times of the wavelength) appears in three consecutive fields (three images). If the fields are viewed successively, the stripes will appear as if they were flowing vertically.
While FIGS. 3 and 4 illustrate only a lightness change due to the flicker, the image quality will in reality be remarkably degraded because a color change additionally takes place. Particularly, the color flicker phenomenon becomes remarkable particularly as the shutter speed rises and the influence of the phenomenon appears in the image picked up by an XY address scanning type imaging element to further degrade the image quality.
If the exposure time can be set to integer times of the cycle period ( 1/100 seconds) of the luminance change of the fluorescent lamp for an XY address scanning type imaging element, the exposure value is held to a constant level so that no flickering of fluorescent lamp, including flickering in the image, takes place regardless of the timing of exposure.
However, if the electronic shutter speed is made variable for a CMOS imaging element, the image pickup apparatus including the CMOS imaging element will become structurally complex. Additionally, if it is not possible to set the exposure time equal to integer times of 1/100 seconds for the purpose of preventing flicking from taking place in an image pickup apparatus having a high degree of freedom for operating an electronic shutter, the degree of freedom of operation of the exposure value adjusting unit for obtaining an appropriate exposure is inevitably reduced.
Thus, there have been proposed techniques of reducing the flickering of fluorescent lamp that is specific to XY address scanning type imaging elements such as CMOS imaging elements.
Patent Document 1 (Jpn. Pat. Appln. Laid-Open Publication No. 2000-350102) and Patent Document 2 (Jpn. Pat. Appln. Laid-Open Publication No. 2000-23040) describe methods of estimating the flicker component by metering the quantity of light of a fluorescent lamp by means of a light receiving element or a light metering element and controlling the gain of the video signal from an imaging element according to the result of the estimation.
Patent Document 3 (Jpn. Pat. Appln. Laid-Open Publication No. 2001-16508) described a method of picking up two images of different types by means of an imaging element under two different conditions, including a first conditions of a first electronic shutter value suitable for the current external lighting condition and a second condition of a second electronic shutter value having a predetermined relationship with the flickering cycle period of a fluorescent lamp, estimating the flicker component by comparing the signals of the two images and controlling the gain of the video signal from the imaging element according to the result of the estimation.
Patent Document 4 (Jpn. Pat. Appln. Laid-Open Publication No. 11-164192) describes a method of storing information on how the lightness changes under the lighting of a fluorescent lamp in a memory in advance as a correction coefficient, while detecting the phase of the flicker component from the video signal obtained from an imaging element by utilizing the difference of frequency between the video signal component and the flicker component, and correcting the video signal by means of the correction coefficient stored in the memory according to the result of the detection.
Patent Document 5 (Jpn. Pat. Appln. Laid-Open Publication No. 2000-165752) describes a method of computationally determining a correction coefficient from two video signals obtained by two exposures carried out with a time difference for inverting the phase of flickering exactly by 180° and correcting the video signal from an imaging element by means of the computationally determined correction coefficient.
However, the methods of estimating the flicker component by metering the quantity of light of a fluorescent lamp by means of a light receiving element or a light metering element as described in the Patent Documents 1 and 2 increase the size and the cost of the system of the image pickup apparatus because a light receiving element or a light metering element has to be added to the image pickup apparatus.
The method of picking up two images of different types under two different shutter conditions and estimating the flicker component as described in the Patent Document 3 is also accompanied by a drawback of making the system of the image pickup apparatus a complex one. Additionally, this method is accompanied by an additional drawback of not suitable for picking up moving images.
The method of using the prepared coefficient that is stored in a memory as correction signal as described in the Patent Document 4 is accompanied by a drawback that it is not possible to reliably detect and reduce the flicker component depending on the types of fluorescent lamp because it is not possible to prepare correction coefficients for all the types of fluorescent lamp. Additionally, with the method of the Patent Document 4 of extracting the flicker component from the video signal, utilizing the difference of frequency between the video signal component and the flicker component, it is difficult to detect the flicker component, discriminating it from the video signal component in the black background part and the low illuminance part of an image where the flicker component is very small, and the performance of detecting the flicker component is degraded remarkably when one or more than one moving objects exist in the image.
Finally, the method of picking up images of two different types at different timings to estimate the flicker component as described in the Patent Document 5 makes the system of the image pickup apparatus complex and is not suited for picking up moving images as in the case of the method of the Patent Document 3.
As described in Patent Document 6 (Jpn. Pat. Appln. Laid-Open Publication No. 2004-222228), the applicant of the present patent application proposed a method of highly accurately detecting the flicker of a fluorescent lamp that is observed specifically by XY address scanning type imaging elements such as CMOS imaging elements by means of a simple signal processing operation without using a light receiving element regardless of the subject, the video signal level and the type of fluorescent lamp to reliably and satisfactorily reduce the flicker.
With the flicker reducing method of the above cited document, the signal components other than the flicker component is removed with normalized integral value or normalized difference value so that, regardless of the subject, it is possible to obtain a signal with which the flicker component can be highly accurately estimated even in the black background part and the low illuminance part of an image where the flicker component is very small. Then, it is possible to highly accurately estimate the flicker component, regardless of the type and the waveform showing a luminance change of the fluorescent lamp, by extracting the spectrum up to an appropriate degree of the normalized integral value or normalized difference value. Thus, it is possible to reliably and satisfactorily reduce the flicker component from the input video signal by computationally operating the estimated flicker component and the input video signal.
Meanwhile, the flicker of a fluorescent lamp in the form of horizontal stripes that arises in XY address scanning type image pickup apparatus shows different characteristics depending on the relation between the frequency of the supplied power and the field or frame frequency of the broadcasting system. For example, it will be found by comparing the characteristics of the stripes that appear in the image that the stripes vary from image to image in terms of frequency, wavelength, waveform and amplitude. It will also be found by comparing a plurality of images among fields that the cyclicity of stripes also varies. FIGS. 5A through 5D illustrate how the cyclicity of stripes varies. When an image pickup operation is conducted by means of the NTSC system under the lighting of a fluorescent lamp in a geographical area where the frequency of the commercially supplied AC is 50 Hz, the flicker stripes show a cyclicity of three fields as shown in FIG. 5A because the imaging cycle period and the power supply cycle period regain the original phase relationship in every three field. Then, when the picked up image is viewed continuously, horizontal stripes appear as if they were flowing vertically. A similar phenomenon appears when an image pickup operation is conducted by means of the PAL system under the lighting of a fluorescent lamp in a geographical area where the frequency of the commercially supplied AC is 60 Hz. In this case, flicker stripes appear as if they were flowing vertically with a cyclicity of five fields as shown in FIG. 5B. The technique proposed in the Patent Document 6 (Jpn. Pat. Appln. Laid-Open Publication No. 2004-222228) separates the image component and the flicker component by utilizing the cyclicity of flicker stripes that appear among fields and extracts the flicker component from the frequency spectrum thereof to correct the gain by using the extracted flicker component. This technique is very effective for the problem of flickering of fluorescent lamp that shows such “inter-field cyclicity”.
On the other hand, no flicker stripe appears when an image pickup operation is conducted by means of the NTSC system under the lighting of a fluorescent lamp in a geographical area where the frequency of the commercially supplied AC is 60 Hz or by means of the PAL system under the lighting of a fluorescent lamp in a geographical area where the frequency of the commercially supplied AC is 50 Hz. Horizontal stripes of flicker appear by the same token when the shutter is operated at high speed with either of the above listed combinations of the broadcasting system and the power supply frequency. However, with either of the above listed combinations, the relative relationship between the imaging cycle period and the power supply cycle period is conclusive within a single field as shown in FIGS. 5C and 5D so that, when the picked up image is viewed continuously, horizontal stripes do not appear as if they were flowing vertically. In other words, flicker stripes that arise with either of the above listed combinations differ remarkably from flicker strips that arise with the combinations described in the preceding paragraph because they constantly appear at the same positions. Thus, with the algorithm of the technique described in the Patent Document 6 (Jpn. Pat. Appln. Laid-Open Publication No. 2004-222228) that utilizes the “inter-field cyclicity”, it is no longer possible to discriminate flickers from the subject with either of the combinations. Then, it is no longer possible to remove or alleviate flickers with that technique. Many flicker correcting techniques that have hitherto been proposed utilize the “inter-field cyclicity” but are accompanied by the same problem that their algorithms are reduced totally ineffective with those combinations.
This problem can be avoided by tactfully selecting the shutter speed. No flicker stripes appear when n/100 seconds (where n is a natural number) in the case of FIG. 5A or 5C or n/120 seconds in the case of FIG. 5B or 5D is selected for the shutter speed regardless of the timing of exposure. The relationships between the shutter speed and the frequency of the supplied power are summarily listed in FIG. 6.
However, the above-described arrangement reduces the degree of freedom of the exposure adjusting unit for realizing an appropriate exposure because the shutter speed is fixed to n/100 seconds or n/120 seconds. It will not be allowed to discard the freedom of exposure adjustment simply to avoid the flicker problem. There arises an additional problem that some unit is necessary to estimate the frequency of the supplied power. For example, the frequency of the supplied power may be determined from the relationship between the cycle period of flicker stripes and the frame rate. However, when the flicker stripes do not move, it is not possible to determine if the stripes appearing in the video signal are produced by flickering or belong to the subject. While this problem can be avoided by using an external sensor, the use of such a sensor is accompanied by a problem of cost and size. Additionally, since the “inter-field cyclicity” is determined according to information coming from the image pickup apparatus, a problem of post-detection can arise when information on the angle of view changes due to framing or some other reason. Then, the most reliable technique for avoiding flicker stripes may be that of fixing the shutter speed to n/100 seconds or n/120 seconds rather than the technique of the Patent Document 6 (Jpn. Pat. Appln. Laid-Open Publication No. 2004-222228).